Repository navigation
Conversation
Sorting a where query by a property of an association, such as sort: 'club.name', failed with a QueryException when the query also had criteria on that association. The criteria join the association under a generated alias, and the order was added with the association path, which Hibernate cannot resolve. The order now uses the alias of the join.
jdaugherty
left a comment
There was a problem hiding this comment.
Verified locally on 0676c67 with :grails-data-hibernate5-core:test. The spec passes, and probes beyond it pass as well: a paged list(max: 2) with totalCount, max plus offset descending, .order('club.name', 'desc') on the detached criteria, a disjunction that uses the association (club.name == 'Arsenal' || name == '...'), a multi-key sort map mixing club.name and name, an explicit alias in the where query combined with a sort by club.name, and ignoreCase: false. The paged count works because HibernateQuery.clone() copies the criterion entries and the subcriteria but not the orders, so the count query carries no order by.
The change is right: getOrCreateAlias registers the join under both the association path and the alias, so resolving the order through criteriaAndAlias.alias covers the generated alias, the explicit alias and the association id in one place, and the detachedCriteria branch closes a latent NPE the old code had for a subquery-backed query. I also confirmed that a sort by an alias the query does not define is silently ignored (the else branch finds no such property), as the docs now say.
One thing to point out: the sort resolves a single association level only. A nested path such as team.club.name still throws, whether or not the where query joins it, while the new docs sentence reads as if any association path works. Details inline on order(). The other inline note is a merge-up heads-up for the Hibernate 7 docs on 8.0.x.
| Without an explicit alias, sorting on association properties may fail due to dynamically generated aliases. | ||
|
|
||
| For example the following query will fail: | ||
| To sort by a property of an associated entity, use the association path. This also works when the `where` query uses the same association: |
There was a problem hiding this comment.
Merge-up note: on 8.0.x the Hibernate 7 copy of this guide (grails-data-hibernate7/docs/src/docs/asciidoc/querying/whereQueries.adoc, section "Query Aliases and Sorting") still shows the owner.lastName sort as failing because of a dynamic alias, while the Hibernate 7 WhereQueryWithAssociationSortSpec there already expects it to work. The merge-up carries only this file, so that section stays stale unless it is aligned in the merge-up or a follow-up.
There was a problem hiding this comment.
Thanks. That section was never updated for Hibernate 7. I'll align it in a separate PR against 8.0.x, after checking the Hibernate 7 behavior a bit beyond what its spec covers.
| if (createdAssociationPaths.containsKey(sortHead)) { | ||
| // sortHead may be either an explicit alias or an association path that | ||
| // was joined (with a generated alias) by a where query criterion. | ||
| // Hibernate can only resolve the property through the alias. |
There was a problem hiding this comment.
This resolves one level only: property is split at the first dot, so a sort by a nested path such as team.club.name (Player -> Team -> Club) still throws a QueryException, whether or not the where query joins team.club. It threw before this change too, so it is not a regression, but the new docs sentence ("To sort by a property of an associated entity, use the association path") reads as if any depth works. Either extend this to nested paths, or state the one-level scope in the docs and leave nested paths to a follow-up.
There was a problem hiding this comment.
Right, it's one level only, as before. I've stated that in the docs now and would leave nested paths for a follow-up. Supporting them means walking the path, reusing the joins that the criteria register under the full path (such as team.club), and adding inner joins for the rest, which I'd rather not do on 7.0.x.
Explicit aliases don't help with nested paths either: the where DSL doesn't recognize def c = team.club, or def t = team; def c = t.club, as an alias, so a query that uses c doesn't compile.
There was a problem hiding this comment.
I'm actually going to fix this ...
There was a problem hiding this comment.
Pushed c0c5a80 to this branch. It reverts b494caa and makes order() walk the sort path one association at a time: a join the criteria registered under the full path, such as team.club, is reused, and each association not joined yet gets an inner join, so team.club.name sorts whether or not the where query joins it, and t.club.name works through an explicit alias of team. A sort by two properties of the same association no longer raises duplicate association path either, since the sort join is registered instead of created for every order. WhereQueryWithAssociationSortSpec covers the nested and duplicate cases (all six failed before the change), and the docs sentence now says the path can cross more than one association.
There was a problem hiding this comment.
@matrei I think it's worth doing this after looking at it. Take a look at the change and see what you think.
There was a problem hiding this comment.
Thanks, that looks good to me, and it brings Hibernate 5 in line with Hibernate 7, which already sorts by nested paths. I ran the grails-data-hibernate5-core tests on c0c5a80 locally, since CI didn't run them after I removed run ci, and they all pass.
Comparing a few queries before and after the change, the results are the same except for one: with fetch: [owner: JoinType.LEFT] on a nullable association, a sort by owner.name now keeps the rows without an owner, because the sort join uses the requested join type instead of an inner join. I think that's right, but since it changes results on 7.0.x, I'll mention it in the PR description.
A sort by a path through more than one association, such as owner.employer.name, is still not supported.
This reverts commit b494caa.
A sort by a path that crosses more than one association, such as team.club.name, failed with a QueryException whether or not the where query joined the nested association, because order() split the path at its first dot only. The sort property is now qualified with the alias of the last association on its path: an association the query has already joined is reused, and one it has not joined yet is joined for the sort and registered like any other join. A sort by two properties of the same association no longer fails with "duplicate association path" either, since the sort join is registered instead of created anew for every order.
matrei
left a comment
There was a problem hiding this comment.
I reviewed c0c5a80 and it looks good to me. Walking the path one association at a time and going through getOrCreateAlias reuses the joins the where query made. It also registers the sort's own join, which fixes the duplicate association path error for two sorts on the same association.
Two minor points, neither blocking:
- No test covers the
LEFTfetch join now being used for the sort. - The docs could say that a sort joins an association with an inner join unless
fetchasks for another type, so a sort byowner.nameleaves out pets without an owner. That was already the case before this change.
jamesfredley
left a comment
There was a problem hiding this comment.
Sorting by an association path reuses a join the where query already made and orders by that join's alias. A path through more than one association works the same way. Not approving. A sort by a nested identifier, such as team.club.id, inner-joins the last association and drops rows whose association is null. Those rows were returned before, because the identifier was ordered through the foreign key.
Since order() walks the whole sort path, a sort by the identifier of an association that follows another one on the path, such as team.club.id, joined that last association too. The join is an inner join unless fetch requests another type, so the sort left out the rows where the association is null. Before, only the first association was joined and Hibernate read club.id from the foreign key of team, which kept them. The identifier is sorted by the foreign key again when the entity before the association holds it. An association without such a foreign key, a hasOne or the inverse side of a one-to-one, is still joined, and so is the first association of a path, as it always has been. The spec also covers the join type of a sort join: an inner join by default, and a left join when fetch requests one. Team.club is nullable in the fixture for these cases. The docs describe the inner join, how to request a left join, and the identifier of a nested association.
jdaugherty
left a comment
There was a problem hiding this comment.
Pushed 5e71443 for the feedback on c0c5a80.
@jamesfredley: a sort by a nested identifier keeps its rows again. team.club.id is sorted by the foreign key on team, as it was on 7.0.x, and club is joined only when there is no such foreign key. Details are in the thread.
@matrei: both points are in the same commit.
WhereQueryWithAssociationSortSpecnow covers the join type of a sort join: an inner join by default, which leaves out a team without a club, and a left join withfetch: [club: JoinType.LEFT], which keeps it.Team.clubis nullable in the fixture for that.- The docs say that an association is joined with an inner join unless the query requests another type, show the
fetchexample, and mention the nested identifier.
An association that a where query joined only for a sort was joined with an inner join, so the sort left out the rows where the association is null: a sort by club.name or club.id dropped the teams without a club, from the list and from the total count of a paged list. A sort must not change which rows a query returns, so the association is now joined with a left join, unless the query requests another join type for it. That keeps the rows for every association on a sort path, so the identifier of a nested association no longer needs to be sorted by its foreign key, and that special case is removed. An association that the where query has criteria on is still joined as the criteria join it, which is an inner join by default.
list, find and findAll with an example, findWhere, findAllWhere and a criteria query that is passed pagination arguments joined an association for a sort with an inner join, as where queries did. A sort by club.name left out the teams without a club, findAllWhere did not find such a team at all, and the total count of a paged list did not count it. These sorts now join the association with a left join too, so a sort no longer changes which instances a query returns. Dynamic finders sort through the same code as where queries and already keep those instances. The new spec covers them as well. The upgrade notes describe the change and how to leave the instances out as before.
jdaugherty
left a comment
There was a problem hiding this comment.
Follow-up to my review above: 2154153 and 1b70845 replace the approach of 5e71443.
A sort should not change which rows a query returns, so an association that is joined only for a sort is now joined with a left join. That keeps the rows for every association on a sort path, so the foreign-key ordering for a nested identifier from 5e71443 is removed again.
@matrei: this goes beyond the two points in your review and changes results on 7.0.x, so I have updated the PR description and added section 22 to the upgrade notes.
- Where queries, detached criteria and dynamic finders (2154153): without a requested join type the sort join was an inner join, and a sort by
club.nameorclub.idleft out the teams without a club, from the list and from thetotalCountof a paged list. They are kept now. A join type requested for the association, such asfetch: [club: JoinType.INNER], is used for the sort, and an association the query has criteria on is still joined by the criteria.calculateProjectionPropertyNamein the same class already left-joins the associations it joins for a projection, for the same reason. list,findandfindAllwith an example,findWhere,findAllWhereand a criteria query with pagination arguments (1b70845) had the same inner join inGrailsHibernateQueryUtilsandGrailsHibernateUtil.Team.list(sort: 'club.name')left out the teams without a club, andfindAllWherewith that sort did not find such a team at all. They use a left join now too.- The where query docs say that sorting does not change which rows a query returns.
WhereQueryWithAssociationSortSpecand the newAssociationSortSpeccover it, and nine cases of each fail on c0c5a80.
@jamesfredley: the details for team.club.id are in the thread.
A criteria query that is passed pagination arguments and a sort by a path through more than one association, such as team.club.name, failed when the total count was read: the sort joined each association from the criteria of the one before it, and the count recreates every join from the root criteria by its path, so it looked for club on the root entity. When the criteria already used the first association of the path, the query failed with "duplicate association path" instead. The sort now reuses a join that the criteria have for an association path, and creates any other from the root criteria with its whole path.
|
@jamesfredley @matrei @borinquenkid, a question that came up while testing this PR and that the PR does not change: what should the With four teams, one of which has no club (
"Fetched" is So in
It also means that The options I see:
What do you think should happen, and on which branch? The two rows where Hibernate 7 does not fetch look like a separate gap there. |
list with a map of sort properties sorted by the last entry only: each entry was passed to CriteriaQuery.orderBy, which replaces the orders of the query. Each order is now added to the ones the query has.
jamesfredley
left a comment
There was a problem hiding this comment.
A sort by a nested identifier such as team.club.id now left-joins that association and keeps the rows whose association is null. A paged criteria count can sort by a nested path, and each entry of a sort map is kept. Not approving. A sort stored on the query is applied before the list arguments, so an inner join requested there does not replace the left join the sort already created.
A sort joined an association as soon as the sort was added to the
query. A sort stored on a where query is added before the arguments of
list are read, so the join was already a left join when fetch requested
another type:
Team.where { ... }.sort('club.name').list(fetch: [club: JoinType.INNER])
returned the teams without a club. Criteria passed to list are added
after a stored sort too, and reused its left join instead of the inner
join they get without the sort.
A sort now only reserves the alias of an association that nothing else
in the query has joined, and the join is made when the query is
executed, with the join type the query requests for the association by
then, or else a left join. Criteria or a projection that need the
association before that join it themselves, under the reserved alias.
A where query, a detached criteria or a dynamic finder can request a
join type for an association, with fetch: [club: JoinType.INNER] or
join('club', JoinType.INNER). The type was only used when criteria or a
sort joined the association. Otherwise the association was fetched with
the outer join of its fetch mode, so the query returned the instances
without the association, and adding a sort to it changed the rows it
returned.
A requested join type other than a left join is now applied when the
query is executed, by joining every association on the requested path
that the query has not joined. Hibernate fetches a collection only
through an outer join, so an inner join requested for a collection
keeps the outer fetch join and restricts the query to the rows whose
collection is not empty.
The fetching guide describes the join types, and section 23 of the
upgrade notes the change in results.
The criteria builder accepted a join type, in join(String, JoinType) and as a JoinType in the fetch argument of list, and did not use it. join set a fetch mode, which Hibernate joins with an outer join unless the association is required, and a JoinType in the fetch argument matched no fetch mode, so the association was not fetched at all. Both now join and fetch the association with the requested join type, as a where query does. The join types requested with join are applied once the criteria are built, so that an association that the criteria have a block for keeps the join of that block. An inner join requested for a collection keeps the outer fetch join and restricts the criteria to the rows whose collection is not empty. The fetching guide and section 23 of the upgrade notes now cover the criteria query.
The subquery that restricts a query to the instances that a requested join returns compared its own fixed alias with the root alias. With a detached criteria named join_subquery or join_subquery0 it compared the subquery with itself, so the restriction matched every row. The alias is now made from the alias of the root, which it can never equal.
A sort through an association that the list fetches joins from the fetch. The count drops the fetches, and that join with them, so Team.list(max: 2, sort: 'club.fans.name', fetch: [club: 'join']) paged through 7 rows with a total of 3. The fetch and what is joined from it are now joined again for the count, under aliases of their own: the root keeps the alias from rendering the list, and Hibernate would generate the same one for a new join.
The check only looked at the collections of the queried class, so a bag fetched through an association, as with publisher.awards, or through a self-reference, as with parent.children, was still read from the repeated rows of the sort. Both kinds of query now follow what the query fetches and compare by path.
Making the join of the criteria an inner join lost the instances without
an element when the criteria accept that, as players { kind == null }
does, which 7.0.x returned. The left join stays. The other collections
that the query fetches are now initialized once the query has run, with
a select for each instance, so that Hibernate does not read them from
the columns of the join of the criteria.
This applies when the query requests a left join of a collection that
its criteria use. The upgrade notes and the fetching guide no longer
describe an inner join.
jdaugherty
left a comment
There was a problem hiding this comment.
Pushed four commits for the review of 8c4df83. The head is a5bfa75.
- a5bfa75: a requested left join stays a left join, so the squad without players is returned. See the thread on
AbstractHibernateQuery. - 01ede72: the alias of the subquery is made from the root alias. See the thread on
GrailsHibernateUtil. - a6346ce, for the count in your summary: a sort join under a fetch is joined again for the count.
Team.list(max: 2, sort: 'club.fans.name', fetch: [club: 'join'])paged through 7 rows with a total of 3, and now has a total of 7, the same as without the fetch. A team without a club still counts, andJoinType.INNERleaves it out. - bdd9a85, for the bag in your summary: the exception now follows what the query fetches, at any depth, and compares by path. A sort by a collection next to
fetch: ['publisher.awards': JoinType.LEFT]orfetch: ['parent.children': JoinType.LEFT]throws when either collection is a bag, inlistand in the other kinds of query. The same fetch without a sort by a collection holds each element once.
AssociationSortSpec and AssociationJoinTypeSpec cover the changes, and 20 of their cases fail on 8c4df83. The whole test suite of grails-data-hibernate5-core passes, and so does codeStyle.
jamesfredley
left a comment
There was a problem hiding this comment.
A requested left join is no longer turned into an inner join when another collection is fetched. Not approving. A scalar projection of a bag still throws, a mapping fetch below the root still repeats a bag, and a map-valued nested fetch throws.
A query that projects values loads no collection, so a property of a bag next to a fetched collection threw where 7.0.x returned the values. The check now leaves a projection alone, in every kind of query. A fetch that a mapping declares below the root was not looked at, so a bag that it fetched was read from the repeated rows of a sort. The check now follows those fetches too, and stops at a mapped association that is already on the path, so that a class that fetches itself comes to an end.
A fetch path through a map handed the map itself to the reflector, which threw an IllegalArgumentException.
jdaugherty
left a comment
There was a problem hiding this comment.
Pushed two commits for the review of a5bfa75. The head is 056588e.
- 0d97b4c: a query that projects values is no longer rejected. See the thread on
GrailsHibernateUtil. - 0d97b4c, for the mapping fetch in your summary: the exception now also follows the fetches that a mapping declares below the root, so a sort by a collection next to a bag that
fetch: 'join'in a mapping loads throws, as it does for a bag that the query asks to fetch. It stops at a mapped association that is already on the path, so a class that fetches itself comes to an end.listis not affected, because it does not join what a mapping fetches, andfetch: ['publisher.awards': 'select']turns the mapped fetch off for a query. - 056588e, for the map in your summary: a fetch path through a map now goes through its values, so the collections on the path are loaded after the query.
AssociationSortSpec and AssociationJoinTypeSpec cover the changes, and 13 of their cases fail on a5bfa75. The whole test suite of grails-data-hibernate5-core passes, and so does codeStyle.
jamesfredley
left a comment
There was a problem hiding this comment.
A scalar projection of a bag no longer throws, and a map-valued nested fetch no longer throws. Not approving. A sort by a collection that the mapping already fetches still throws, and a sort that names the root alias throws.
A sort by a bag that the mapping fetches with fetch: 'join' threw next to a fetched set or list, because only a fetch that the query requests counted as a fetch of the sorted collection. The query fetches such a collection anyway, so the sort makes no join of its own and the query returns the collections that it returns without the sort. It is now only rejected next to another bag, as for a requested fetch. list does not fetch what the mapping fetches, so its sort does join such a collection without reading it. It accepts the sort as well now, unless the collection that it fetches is a bag, which the rows of that join would repeat.
new DetachedCriteria(Team, 't').list(sort: 't.name') threw Invalid sort property, and 7.0.x ignored the sort. A sort whose first part is the alias of the root of the query, and not a property of the queried class, now sorts by the rest of its path, in where queries, detached criteria, dynamic finders, find and findAll with an example, findWhere and findAllWhere, and the sort argument of a criteria query. The sort argument of a criteria query ignored a sort by an alias that the criteria do not define. It now throws Invalid sort property, as the other queries do.
jdaugherty
left a comment
There was a problem hiding this comment.
Pushed two commits for the review of 056588e. The head is 7323d92.
- 3677b3a: a fetch from the mapping counts as a fetch of the sorted collection. See the thread on
GrailsHibernateUtil. - 7323d92: a sort can start with the alias of the queried class. See the thread on
AbstractHibernateQuery.
28 cases of AssociationSortSpec and UndefinedAliasSortSpec fail on 056588e. The whole test suite of grails-data-hibernate5-core passes, and so does codeStyle.
✅ All tests passed ✅🏷️ Commit: 7323d92 Learn more about TestLens at testlens.app/docs. |
|
Thanks for all the work on this. The PR has grown from the fix for #15589 into a rework of how GORM for Hibernate 5 joins and fetches associations: 53 commits, 24 files and about 6,200 added lines, and sections 22 and 23 of the upgrade notes list the behavior changes for 7.0.18. I targeted 7.0.x because the fix was backwards compatible, and most of the PR isn't anymore. Queries return and count other rows than before, I'd like to split it:
If you agree, I'll push the current head to a new branch for the new PR, and then reset this branch to 5e71443. |
jamesfredley
left a comment
There was a problem hiding this comment.
Not approving. A sort by an alias is dropped, and a mapped fetch is treated as joined when the fetch depth is 0.
| // embedded objects cannot reference entities (at time of writing), so the rest is a component path | ||
| break; | ||
| } | ||
| if (!(persistentProperty instanceof Association)) { |
There was a problem hiding this comment.
list(max: 1, sort: ['c.name': 'desc', name: 'asc']) { createAlias('club', 'c'); like('c.name', '%') }, with A in Arsenal and B in Chelsea. The list returns A. The sort by c.name is dropped. Keep that sort.
| } | ||
| String path = prefix + propertyNames[i]; | ||
| FetchMode requested = getRequestedFetchMode(root, path); | ||
| boolean joined = requested != null ? requested == FetchMode.JOIN : |
There was a problem hiding this comment.
hibernate.max_fetch_depth is 0, and group.awards is a bag. Owner.where {}.list(sort: 'children.name') throws. The bag is not joined. Do not treat it as fetched.
|
On Two corrections to what I first wrote here: the upgrade note for it is already in this PR (item 23, the bullet on @matrei, when you split this PR: would you keep this commit, with those specs, on 7.0.x? If it goes to 9.0.x with the rest, 8.0.1 ships with Hibernate 5 and Hibernate 7 returning different rows for that call, and #16939 needs another answer. |
|
@matrei I agree the PR has become too big. I'd like to propose a different cut: by bug, not by branch. Every part of it fixes a query that returns the wrong rows, modifies the wrong rows, fails, or silently ignores what it was asked, on 7.0.x today. A bug is fixed on the oldest branch that has it and merged up, so each of the PRs below would target 7.0.x.
A fix for a wrong result always changes that result. What we owe applications is an upgrade note that says what changed and how to get the earlier result, and each PR below would carry its own bullets from sections 22 and 23.
That would leave these bugs in every 7.x and 8.x release, including a
Agreed, and that is what the split is for. Nearly all of those findings are in the rules for collections, so those are a PR of their own at the end (number 10), and the rest does not have to wait for it. The splitEach PR is one bug, with its own specs, docs and upgrade note. They are stacked in this order. "7.0.x" is what the branch does today.
Does this cut work for you? If you see one of these as something other than a bug on 7.0.x, tell me which, and we can look at that one by itself. If it does, I'll reset this PR & split these off. |
@matrei @jdaugherty I'm for the cut by bug, each piece against 7.0.x. My reasons:
Each piece should carry three things:
Pieces 5 and 10 add a new For ordering: I have open PRs on the same files (#16890, #16891, #16930, #16931, #17008, #17014). I will rebase them onto whichever of these pieces lands first and won't touch the sort or |
|
@jdaugherty thanks, splitting by bug is better than my cut. A single 9.0.x PR with the other 5,900 lines would have the same problem this one has. One PR per bug, stacked, each with its own specs, docs and note: agreed. Item 1 as this PR, reset to 5e71443: agreed, with one change that I describe below the table. Where we still differ is the branch, and I don't think "every item is a bug" settles it. The template says patch branches take bug fixes only, but it doesn't say whether a fix that changes the result of a query that runs today counts as one. I'd use this test for each item:
Why I don't think an upgrade note is enough on 7.0.x:
Applied to your list:
Item 1 at 5e71443 doesn't fully pass that test. Without criteria on the association, a sort by it works on 7.0.x and joins it with an inner join, even when the query requests a left join. 5e71443 uses the requested join type for the sort, so a query that works today returns more rows: // Rex (owner Bob), Tom (owner Ann), Stray (no owner)
Pet.where { name ==~ '%' }.list(sort: 'owner.name', fetch: [owner: JoinType.LEFT])*.name
// 7.0.x: [Tom, Rex]
// 5e71443864: [Stray, Tom, Rex]The new result is the better one, since the same query without the sort returns Stray, but it's the change of item 2. So after the reset I'll make a join that only the sort needs an inner join again, as on 7.0.x, and take the Two more things that point the same way: The change in 1e47513 and #16898. @borinquenkid, I'd rather revert #16898 on 8.0.x before 8.0.1 is cut, keep it on 9.0.x when the revert is merged up (it's already there), and land 1e47513 with item 8 on 9.0.x. Then Hibernate 5 and 7 return the same rows in 8.0.1, as in 8.0.0, and the same rows in 9.0, and #16940 becomes a 9.0.x note. If we keep #16898 in 8.0.1 anyway, 1e47513 should go to 8.0.x on its own rather than to 7.0.x: 7.0.x has no Hibernate 7 to agree with. It's a three-line change, and the 13 cases of For the 9.0.x part:
If this works for you, please push 7323d92 to a branch before resetting, so nothing is lost. Since the same branch question came up on #16898 and #16939, I'll also start a thread on dev@ so we settle it once and don't argue it per PR. |
|
@matrei thanks, we agree on more than we differ on: one PR per bug, stacked, each with its own specs, docs and note; a regression of a PR is fixed in it, and an older issue that a review finds gets an issue of its own. I also agree with your change to item 1, see below. On the branch:
That test protects an application whose query runs today. It does not ask whether the query does what it was asked, and for these items it does not:
The policy we have is the one in the template: patch branches take bug fixes. A bug fix to a query that runs always changes what that query returns, so a test that excludes those excludes most fixes to queries. Your own table shows that: items 9 and the root alias of 11 change the result of queries that run today, and you put them on 7.0.x, rightly.
Agreed, the place is wrong. That is a reason to move the notes, not the fixes: a section for changes within 7.0.x at the top of the upgrade guide, and each PR titled so that the drafted release notes name the change. I'll do that in the split.
Section 4.4.4 is about path expressions in a JPQL query. A
Each new exception replaces a wrong result, not a right one: repeated elements in a bag, and an update or delete of rows that the query does not return. I take the point on the query shapes, and each PR should say in its description which SQL changes, so a reviewer can weigh it.
It holds for a bug. A feature, or a deliberate change of a defined behavior, has no less correct side, and those wait for a major. Moving the rest to 9.0.x also skips 8.x: Hibernate 5 ships in 8.0.x and 8.1.x, and they would keep all of this until 9. Where you are right:
On 1e47513 and #16898: @borinquenkid and I would keep both where they are, for the reason above. Where it stands: the branches are cut locally, each on top of the one before it, and the last one has the tree of 7323d92 plus fixes for both findings of review 5868034911 (the alias sort in a paged criteria query in the criteria sort PR, A thread on dev@ is the right place to settle the branch question. |
|
@jdaugherty thanks, that's a lot of common ground, and agreed that dev@ settles the branch question: https://lists.apache.org/thread.html/lsg5glbf0f4xjsppm521c7vw9yrh64r0. So here only what doesn't depend on it. The
The first query agrees with itself on 7.0.x today, because On items 9 and 11: they change the order of rows that the caller asked to have sorted, not which rows come back. That's the line in my test, and the one I proposed on dev@. The order of the stack. Several pieces are on 7.0.x in both of our tables:
Could those go first in the stack, before item 2? Then they can be reviewed and merged now, whatever dev@ decides, and items 2 to 8, which have to be released together anyway, follow as one block. For item 7 that means sorting with the join that the Item 4 and the merge-up. A merge-up can't leave a commit behind. If item 4 goes to 7.0.x, the next merge-up into 8.0.x either waits for Hibernate 7 or brings the change to Hibernate 7 without its spec. So the Hibernate 7 rule and its spec need to be on 8.0.x before item 4 merges into 7.0.x. Agreed:
I'll make the item 1 change after the reset. |
|
@matrei you're right about The one place I found where they disagree on 7.0.x is a stored sort: On the merge-up: agreed, a merge-up can't leave a commit behind. The Hibernate 7 rule and its spec go to 8.0.x before item 4 merges into 7.0.x. On the order of the stack: I'll move the sort map (item 9) and the |
Description
Fixes #15589.
Sorting a where query by a property of an association failed with a
QueryExceptionwhen the query also had criteria on that association:The criteria join
clubunder a generated alias, such asclub_alias0, andAbstractHibernateQueryrecords the join under both the association path and the alias.order()found the join, but added the order with the association path, which Hibernate cannot resolve against the root entity. The sort only worked with an explicit alias defined in the where query, because then the sort property already started with the alias. The order is now added with the alias of the join, asclub_alias0.name.The same sort already worked when the query had no criteria on the association, and the Hibernate 7 version of
WhereQueryWithAssociationSortSpecon 8.0.x already expects it to work.A sort by a path through more than one association, such as
team.club.name, failed whether or not the where query joined the nested association, becauseorder()split the path at its first dot only.order()now walks the path one association at a time. It reuses a join the criteria registered under the full path, such asteam.club, and joins any other association on the path for the sort, registering that join like any other. A sort through an explicit alias, such ast.club.name, works the same way. Because the sort join is registered instead of created anew for every order, a sort by two properties of the same association no longer fails withduplicate association path.Sorting by an explicit alias produces the same order as before, and so does sorting by the association's id, such as
club.id, in a query with criteria onclub, which now goes through the join instead of the foreign key.Sorting no longer leaves out rows
An association that was joined only for a sort was joined with an inner join, so the sort left out the instances whose association is
null.Team.list(sort: 'club.name')did not return the teams without a club,Team.findAllWhere([name: 'A team without a club'], [sort: 'club.name'])did not find such a team at all, and thetotalCountof a paged list did not count them. A sort should not change which rows a query returns, so the association is now joined with a left join. This applies to where queries, detached criteria and dynamic finders, and tolist,findandfindAllwith an example,findWhere,findAllWhereand a criteria query that is passed pagination arguments.fetch: [club: JoinType.INNER]does, is used for the sort as well. Before, the sort ignored a requested join type and always created an inner join.Team.where { ... }.sort('club.name'), is applied before the arguments oflistare read and before criteria passed tolistare added. A join type thatfetchrequests there is therefore used for a stored sort as well, and criteria passed tolistjoin the association with their own join type, as they do without the sort. Those criteria can also join it through an alias, as increateAlias('club', 'c'), which failed after a stored sort, and the sort can be by that alias.orderin a criteria query can sort by a property of an association too, as inTeam.createCriteria().list { order('club.name') }, also by a path through more than one association and inside the criteria of an association. It failed withcould not resolve property, and with aQuery.Orderit sorted by a property of the queried class instead. It joins with a left join, and uses a join that the criteria make for the association. The orders are added once the criteria are built, so an order also uses a join type that the criteria or thefetchargument oflistrequest for its association, and does not keep a collection from being fetched that they ask to fetch, whether the request comes before or after the order. An order by the identifier of an association, such asclub.id, makes no join and reads the foreign key, as it did before this PR, so it can still be combined with a projection that groups by the same path or is distinct on it. Inside the criteria of an association, aQuery.Orderfor a single property is resolved against that association too, as an order by a property name is. An order by the identifier of ahasOne, asorder('contract.id'), is joined with a left join as well: Hibernate has no foreign key to read it from, and joined it with an inner join, also before this PR.team.club.idreturns a player whose team has no club, as it did before this PR, when onlyteamwas joined andclub.idwas read from its foreign key.players.name, fetched the collection, and a row limit left elements out of it: withmax: 2, a team came back with two of its three players. A sort now joins a collection without fetching it, unless the query asks for the collection to be fetched, and the collection is loaded when it is read, as it was before this PR. A collection that the query asks to fetch is fetched with and without the sort. Before this PR, the inner join of the sort took the place of the fetch. When the query fetches another collection, a sort by a collection fetches its collection as well: Hibernate gives a join of a collection that it does not fetch the column aliases of the next collection that it fetches, and fills that one with the elements of the other when both hold elements of the same class. The join of a sort returns a row for each element of the collection, and a bag, a collection declared as aCollection, cannot tell a repeated row from a repeated element, which is why Hibernate does not support fetching more than one bag in a query. So when either collection is a bag, also one fetched through an association or a self-reference, as withfetch: ['publisher.awards': JoinType.LEFT], the query throws anUnsupportedOperationExceptionthat names both collections, inlistas well. On 7.0.x such a query left out the instances whose sorted collection is empty, and a bag that it fetched could hold its elements more than once. A sort by an association to a single instance is not rejected, and neither is a query that asks to fetch the collection that it sorts by, unless both are bags. The condition that keeps a collection from being fetched is part of the where clause, not of the join condition, which Hibernate combines with the filter of a collection that has a join table, such as the tenant filter of discriminator based multi-tenancy, into SQL that no database accepts.This changes results on 7.0.x, so section 22 of the upgrade notes describes it, including how to leave the instances out as before. #16863 makes the same change for Hibernate 7 on 8.0.x, where the two versions then return the same rows for these sorts. Apart from this, only queries that failed before behave differently.
A requested join type is applied without a sort or criteria
A where query, a detached criteria, a dynamic finder or a criteria query can request a join type for an association, with
fetch: [club: JoinType.INNER]orjoin('club', JoinType.INNER). In a where query, a detached criteria and a dynamic finder, the type was only used when criteria, or with this PR a sort, joined the association. Otherwise the association was fetched with the outer join of its fetch mode:Team.where { ... }.list(fetch: [club: JoinType.INNER])returned the teams without a club, and the same query withsort: 'club.name'did not, so a sort still changed the rows of such a query. A requested join type other thanLEFTis now applied when the query is executed, whether or not anything else in the query uses the association.JoinType.INNERleaves out the instances whose association isnull, fromlist,get,find,countand thetotalCountof a paged list, and still fetches the association.JoinType.RIGHTleaves out the same instances and fetches the association. As a right join does, it also has a row for each associated instance that none of the queried instances refers to:listreturnsnullfor such a row andcountcounts it, unless the criteria of the query exclude it.fetch: ['team.club': JoinType.INNER], was ignored without criteria on the path. Every association on it is now joined and fetched. A path through a collection, as infetch: ['players.contract': JoinType.INNER], restricts the query in a subquery, because a join through a collection returns an instance once for each element:countand thetotalCountof a paged list count each instance once. The collection on the path is fetched with outer joins and holds all of its elements, also the players without a contract: the join type decides which teams the query returns and never leaves elements out of a fetched collection, which would leave a partial collection in the session. A where query and a dynamic finder return each team once, and a criteria query and a query by example once for each fetched player, as for any fetch of a collection. AJoinType.LEFTfor such a path fetches the path too..join('club', JoinType.INNER).list(fetch: [club: 'lazy'])returns the teams without a club, as it did before this PR. The same holds forjoin('club')orselect('club')afterjoin('club', JoinType.INNER), on a where query and in a criteria query, and for thefetchargument of a criteria query. The join that criteria on the association make keeps the requested join type, as it did before this PR, soTeam.where { club.name == 'Arsenal' || name == 'Unattached' }.join('club', JoinType.LEFT).list(fetch: [club: 'join'])returns the team without a club.joinor thefetchargument oflistrequests it. Thefetchargument is read after the criteria have been applied, so it was not used for such a join:Team.where { club.name == 'Chelsea' || name == 'Unattached First Team' }.list(fetch: [club: JoinType.LEFT])left out the team without a club.join('team.club', JoinType.LEFT)is used for the join that criteria onteam.clubmake, which was an inner join whatever was requested, and a join type requested forclubis no longer used forteam.club. Criteria on a nested association can also be used more than once, which failed withduplicate association path.join('club', JoinType.INNER)set a fetch mode like the one-argumentjoin, and aJoinTypein thefetchargument oflistmatched no fetch mode, so the association was not fetched at all. Both now join and fetch with the requested type. They are applied once the criteria are built, so an association that the criteria have a block for keeps the join of that block, as does one that the criteria join through an alias, as increateAlias('t.club', 'c').JoinType.LEFTfor it uses that join type for the join of the criteria, and no longer fetches the collection through that join, whose rows only have the elements that the criteria match. Withjoin('players', JoinType.LEFT)the collection was initialized with those elements only, and with thefetchargument it would have been since this PR uses the argument for the join of criteria. The collection is now loaded when it is read, with all of its elements. When such a query also fetches another collection, that collection is initialized once the query has run, with a select for each instance, because Hibernate would otherwise read it from the columns of the join of the criteria when both hold elements of the same class. The left join stays, so the query returns the instances without an element when the criteria accept that, asplayers { kind == null }does.updateAllanddeleteAllapply a requested join type too, so they update or delete the instances thatlistreturns.Team.where { ... }.join('club', JoinType.INNER)listed and counted the teams with a club, butdeleteAllon it deleted the teams without one as well. An update and a delete cannot join, so each requested join is applied to them as a restriction: a subquery that selects from the entity of the first association on the path and joins the rest of it. MySQL, and MariaDB before version 10.3, do not allow an update or a delete to select from the table that it modifies. On these databases a join that leads back to the queried class, such as theparentof a category, is rejected with anUnsupportedOperationExceptionthat says to restrict the query with criteria instead. The check follows the dialect: a MySQL dialect, or a MariaDB dialect older thanMariaDB103Dialect.deleteAllselects the identifiers first when the path starts with a collection that has a join table, because Hibernate deletes the join rows before the instances, with the same restriction, which then finds no instance. A one-to-one whose foreign key the other class has is matched through the property that refers back.list,findWhere,findAllWhere, andfindandfindAllwith an example, apply aJoinTypein theirfetchargument too, before a sort by the association, which uses that join. They ignored it, and since a sort joins with a left join, a sort by the association no longer left out the instances without it either.listfetches the association with the requested join type, returns each instance once when it fetches a collection, and appliesRIGHTlikeINNER, as Hibernate does not support right joins in criteria queries.'join'and'eager'in thefetchargument oflistnow fetch the association with a left join and return every instance, as a where query and a dynamic finder do. They joined it with an inner join without fetching it, so they left out the instances whose association isnullor whose collection is empty.hasMany = [tags: String], too. It was ignored.nullfor each element that no instance has.teaminteam.players.contract, are joined and fetched, and so is the collection.findandfindAllwith an example,findWhereandfindAllWhere. The joins that those criteria need are made after the event; they were left to a query that is not executed, and the query failed.This also changes results on 7.0.x. Section 23 of the upgrade notes describes it, and the fetching guide now documents the join types.
A criteria query that is passed pagination arguments can now also be sorted by a path through more than one association.
Player.createCriteria().list(sort: 'team.club.name', max: 10) { ... }failed whentotalCountwas read, and withduplicate association pathwhen the criteria usedteam. The sort now reuses a join that the criteria have for an association path, and creates any other from the root criteria with its whole path, which is how the count recreates it. A join that the criteria make through an alias, as increateAlias('t.club', 'c'), is reused as well. It can also be sorted through an alias that the criteria give an association, assort: 't.club.name'withcreateAlias('t.club', 'c'), and by a map of sort properties, which failed with aClassCastException. An association after the first one on a sort path, or on a requested join path, is joined through the alias of the join before it, so that Hibernate does not read the path through an alias of the criteria that has the name of its first association.listwith a map of sort properties, such asTeam.list(sort: ['club.name': 'asc', name: 'desc']), sorted by the last entry only, because each entry replaced the orders of the query. It now sorts by every entry. Hibernate 7 on 8.0.x already does. The same holds for a default sort that the mapping declares for more than one property, of whichlistapplied the last one only. An association is joined once for all the sort properties that use it:listjoined it once for each, andfindandfindAllwith an example,findWhereandfindAllWherefailed withduplicate association path.The
totalCountof a paged list counts what the list pages through. A list that sorts by a collection returns a row for each element, andmaxandoffsetpage through those rows, so its total counts them, as on 7.0.x, also when the sort goes through an association that the list fetches. A list that fetches a collection returns each instance once, so its total counts instances, also when a sort joins another collection. A paged query keeps the root alias of a detached criteria when the total is read:new DetachedCriteria(Team, 't').eq('t.name', 'Arsenal').list(max: 2).totalCountfailed on 7.0.x.The "Sorting with Associations" section of the where query docs said that this sort fails and that an explicit alias is required. It now shows sorting by the association path, notes that the path can cross more than one association, that sorting does not change which rows a query returns, and that a sort by an alias that the query does not define fails with
IllegalArgumentException: Invalid sort property, as it does on Hibernate 7. It was ignored.Tests
WhereQueryWithAssociationSortSpecasserted theQueryException. It now checks that a where query with criteria onclubcan be sorted byclub.name:wherecalls andfetch: [club: 'join'].Without the change, these cases fail with the
QueryException. It also checks that the order is as before when sorting byclub.id, by an explicit alias, and byclub.namein a query without criteria onclub.For paths through more than one association, it checks sorting
Playerbyteam.club.namewith criteria onteam.club, with criteria onteamonly, and without criteria on either, and byt.club.namethrough an explicit alias ofteam. It also checks a sort by bothclub.nameandclub.id. These cases failed before the change.For the left join, it checks with a team that has no club that sorts by
club.name,club.id,team.club.nameandteam.club.idkeep the team or its player, also through an explicit alias, with criteria onteam, and in thetotalCountof a paged list. It also checks that a join type requested withfetchis used, and that a criterionclub != nullleaves the team out.Team.clubis nullable in the fixture for that.For a sort stored on the query with
sort, it checks that the team without a club is kept, that a join type requested withfetchinlistorfindis used, also forteam.club.nameand in thetotalCountof a paged list, and that criteria passed tolistreturn the same rows with and without the sort. Seven of these cases fail without the join being made when the query is executed.The new
AssociationSortSpecchecks the same forlist,findAllwith an example,findWhere,findAllWhere, a criteria query with pagination arguments, and a dynamic finder. It also checks a paged criteria query sorted byteam.club.name, with and without criteria onteamand with criteria that jointeam.clubthrough an alias, which failed before, thatlistapplies every property of a default sort, a sort by two properties of one association, andorderin a criteria query, including the sequence of several orders, an order that comes before the criteria or the alias of its association, an order by the identifier of an association together with a projection on it, and aQuery.Orderinside the criteria of an association.Both specs check a sort by a property of a collection as well: that it keeps a team without players, and that it does not fetch the collection, with a row limit and for a single result, so that no player is left out of it, unless
fetchasks for the collection.56 of the 71 cases of
AssociationSortSpec, and 24 of the 59 cases ofWhereQueryWithAssociationSortSpec, fail on c0c5a80, the commit before the left join.The new
AssociationJoinTypeSpecchecks a join type requested without a sort or criteria on the association:'join',LEFT,INNERandRIGHTwithlist, including that the association is fetched, the same rows with and without a sort,count, thetotalCountof a paged list,find,get, a dynamic finder,team.club, theplayerscollection, including that an inner and a right join fetch it, also when the query is sorted by it, a path through the collection, which must not leave elements out of it, a fetch mode requested after a join type, a subquery, also one that has criteria on the association it requests a join type for, and a projection. It checks the same for a criteria query, withjoinand with thefetchargument oflist, includingcount,get, thetotalCountof a paged list, a sort, a collection, a nested path and an association the criteria have a block for or join through an alias. It also checks that a join type thatfetchrequests is used for the join of criteria and of a projection, a join type for a nested association, a fetch mode requested after a join type, and that a path through a collection returns and counts each instance once. For a criteria query that is ordered by an association or a collection, it checks that a join type and a fetch thatjoinor thefetchargument request are applied, before and after the order. It also checks that a left join requested for a collection that a where query has criteria on does not fetch the collection, and thatupdateAllanddeleteAllupdate or delete the instances thatlistreturns, for an association, a collection, a path through either, and an association whose key the associated class has. 101 of its 135 cases fail without the changes. 28 cases of the two specs fail on 9c59ac4, where an order in a criteria query joined its association before the rest of the criteria was known, and nine fail on e8f7036, whereupdateAllanddeleteAlldid not apply a requested join type.The new
MultiTenancyRequestedJoinSpecchecks, with discriminator based multi-tenancy, that an inner or a right join requested for a collection leaves out an instance whose elements all belong to another tenant, from a where query, a criteria query andcount. Six of its seven cases fail on c5b0edb.DetachedCriteriaJoinTypeSpecingrails-datamapping-corechecks that a laterjoinorselectreplaces a recorded join type and keeps it as one that the fetch mode replaced, also in a copy of the criteria.The new
RequestedJoinBulkOperationDialectSpecchecks, with the MySQL and MariaDB dialects, thatupdateAllanddeleteAllreject an inner join requested for an association to the queried class, that the MariaDB 10.3 and H2 dialects apply it, that a join to another class is applied on MySQL, and that criteria on the association restrict the update there. Four of its nine cases fail on cc58a3b.A second review, with each finding run against 7.0.x, added cases for:
deleteAllandupdateAllthrough a collection that has a join table and through the side of a one-to-one that the other class has the foreign key of; a sort by a collection that has a join table and multi-tenant elements, with each kind of query; a sort by a collection while the query fetches another collection of the same class, and the exception when either is a bag; a fetch mode requested after a join type, with criteria on the association; a paged criteria query sorted through an alias, by a map, and by a nested path when an alias has the name of its first association; a join type in thefetchargument oflist,findWhere,findAllWhere, andfindandfindAllwith an example, with and without a sort; a collection of a basic type; the associations before a collection on a path; a right join of a collection that criteria join;order('contract.id')on ahasOne; and a projection into an embedded component of an association. The newPreQueryListenerAssociationCriteriaSpecchecks criteria on an association that a listener adds. 50 of these cases fail on 1ab83ab.WhereQueryWithAssociationSortSpecalso sorts players whose order differs by club name, club id, team name and player name, so that a sort through the wrong join cannot pass.The new
FetchJoinTypeSpecdefines the behavior of a requested join type as a chart: forplayers,players.contractandclubwithINNER,RIGHTandLEFT, which teams the query returns, what it fetches, that a fetched collection holds all of its elements, whether a team is returned once or once for each fetched player, and thetotalCountof a paged list. It checks every row with eight kinds of query: a where query with thefetchargument and withjoin, a dynamic finder, a criteria query withjoinand with thefetchargument,findAllwith an example,listand a pagedlist. 28 of its 64 cases fail on 5e6dc8f.AssociationSortSpechas the same kind of chart for a sort by a set, a list or a bag while the query fetches nothing, a set, a list or a bag, with two elements in each collection, since one element cannot show a repeated row.AssociationJoinTypeSpechas a chart for a left join of a collection that criteria use, with and without another fetched collection, and with criteria that need a value or accept a missing element. It also checks the root alias of a paged detached criteria, including aliases that a subquery could hide.AssociationSortSpecalso checks thetotalCountof a list sorted by a collection and through an association that it fetches, and the exception for a bag fetched through an association or a self-reference. 20 cases of the two specs fail on 8c4df83.These pass with the change:
grails-data-hibernate5-core, including the data TCK, and ofgrails-datamapping-core;codeStyleforgrails-data-hibernate5-coreandgrails-datamapping-core.